ACT Research & Policy | Technical Brief | January 2020 


Promoting Career Readiness in Arkansas 


High Schools 


Jeffrey T. Steedle, Senior Research Scientist, ACT 


Ray Girdler, Director of Data Use & Privacy, Arkansas Department of Education 


Introduction 


The Arkansas Department of Education (ADE) 
is focused on transforming Arkansas to lead 
the nation in student-focused education. To 
accomplish this vision, one of ADE’s goals is 
for each student to meet or exceed milestones 
along pathways to graduate high school 
prepared for college, career, and community 
engagement. Some career and technical 
education (CTE) students in Arkansas high 
schools take ACT® WorkKeys® to measure 
their foundational workplace skills in Applied 
Math, Workplace Documents, and Graphic 
Literacy. Moreover, students can earn a 
National Career Readiness Certificate® 
(NCRC®), which certifies skills to potential 
employers and postsecondary training 
programs.' ADE identified WorkKeys scores 
as a possible data source that could be 
embedded within their electronic transcript 
system (Triand, Inc.) to inform career 
counselors and parents statewide about 
students’ levels of workforce readiness. For 
that reason, ADE partnered with ACT 
Research to share data and explore 
relationships between educational programs 
and foundational workplace skills measured 
by WorkKeys. 


This report focuses on educational programs in 
high schools with greater-than-expected 
WorkKeys performance. That is, compared to 
other schools enrolling students with similar 
ACT scores and demographics, these schools 
exhibited higher average WorkKeys scores. 


Ten such schools were contacted to find out 
how they promote WorkKeys performance and 
career readiness more generally. Results 
revealed common themes such as supporting 
career exploration, connecting students and 
local employers, using WorkKeys Curriculum® 
to prepare students for the WorkKeys 
Assessments, and emphasizing the value of 
the NCRC. 


Method 


Statistical analyses estimated average 
WorkKeys performance relative to expected for 
48 public high schools in Arkansas (see 
Appendix for details of the statistical analysis). 
ADE contacted 10 schools with greater-than- 
expected WorkKeys performance to ask the 
following questions: 


1. Does your school have any special 
programs to help get students career 
ready? 


2. How do your students prepare to take 
WorkKeys? 


3. How do you communicate the value of 
taking WorkKeys and earning a National 
Career Readiness Certificate? 


4. Is WorkKeys recognized by local 
employers or postsecondary training 
programs? 


An administrator, teacher, or counselor from 
each of the 10 schools responded to the 
questions and provided consent to be named 
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in this report. The following Arkansas high 
schools are represented in results: 


e Benton High School 

e Camden Fairview High School 

e Crossett High School 

e Manila High School 

e Pine Bluff High School 

e Rivercrest High School 

e Rogers New Technology High School 
e Russellville High School 

e Warren High School 

e Western Yell County High School 


Results 


Career Readiness Programs 


As part of their programs to support career 
readiness, nearly all schools mentioned the 
Career Readiness, Work Readiness, College 
and Career Readiness, or Jobs for America’s 
Graduates (JAG) course. Taking WorkKeys 
Applied Math, Workplace Documents, and 
Graphic Literacy, which make a student eligible 
to earn an NCRC, is a highly recommended 
component of the standards (or “frameworks”) 
for these courses throughout Arkansas.* The 
Readiness courses focus on developing the 
knowledge and skills as well as social- 
emotional competencies (“soft skills”) needed 
for success in a variety of postsecondary 
pathways. This entails evaluating current 
interests and skills, conducting pathway 
research and planning, making decisions, and 
developing the discipline and dispositions to 
carry out plans and adapt to changes in job 
demands over time. The JAG program shares 
these goals, but it also integrates modest 
interventions to develop reading and math 
skills for students at risk of not graduating on 
time and often belonging to special populations 
(e.g., students with disabilities, economic 
disadvantage, or limited English proficiency). 
At Manila High School, approximately 80% of 
senior students enroll in a Career Readiness or 
Work Readiness course, and they attribute this 


to strong support at the district level and high- 
quality instruction. 


Four survey respondents mentioned that 
students have opportunities to participate in 
internships, work-based learning, or youth 
apprenticeships. At Warren High School, for 
example, students take CTE courses at a local 
community-based education center, and many 
students engage in an internship or youth 
apprenticeship. Russellville High School 
described the integration of a career readiness 
curriculum into CTE courses, as well as close 
collaboration between counselors, teachers, 
and students to develop beneficial Student 
Success Plans. Rogers New Technology High 
School highlighted its broad use of project- 
based learning, which involves problem 
solving, making presentations, and 
demonstrating professionalism. They also 
create opportunities for students to interact 
with local leaders, employers, non-profits, and 
the chamber of commerce to help students find 
mentors and internships. Their goal is to 
prepare all students for college and careers, 
and they see NCRC attainment as validation of 
their approach. 


Preparing for WorkKeys 


Student performance on the WorkKeys Applied 
Math, Workplace Documents, and Graphic 
Literacy assessments is reported using level 
scores that correspond to specific knowledge 
and skills required in a broad range of 
occupations.? WorkKeys Curriculum provides a 
placement exam, instructional modules, 
practice exercises, and quizzes to evaluate 
mastery of Applied Math, Workplace 
Documents, and Graphic Literacy at Levels 3, 
4, 5, 6, and 7. All 10 schools reported using 
WorkKeys Curriculum to help students prepare 
to take the three NCRC Assessments or to 
improve their scores. Indeed, Camden Fairview 
High School asserted that WorkKeys 
Curriculum “prepares them for this very well.” 
Several schools stated that they use WorkKeys 
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Curriculum to determine when students are 
ready to take the NCRC assessments (i.e., to 
“test when they are ready”). For example, at 
Benton High School, students must first pass 
at least the Level 3 exam in the Curriculum. At 
Manila High School, student must complete at 
least Level 5 of the Curriculum. 


Pine Bluff High School mentioned the use of 
classroom instruction focused on skills 
measured by NCRC assessments. Knowledge 
and skills developed through internships, work- 
based learning, or youth apprenticeships may 
also support achievement on WorkKeys 
Assessments. Students may also be better 
prepared for WorkKeys because of 
instructional approaches such as project-based 
learning at Rogers New Technology High 
School. Though not explicitly stated in survey 
responses, academic instruction in English 
language arts, mathematics, science, and 
social studies should be expected to help 
prepare students for the Applied Math, 
Workplace Documents, and Graphic Literacy 
assessments. 


The Value of WorkKeys 


Students exhibit higher motivation on 
achievement tests when they understand the 
importance or value of an assessment. High 
schools responded to this survey with a variety 
of activities they employ to impart 
understanding of the value of an NCRC. For 
example, Pine Bluff High School promotes the 
value of the NCRC by hosting guest speakers 
including industry leaders and representatives 
from the Arkansas Department of Workforce 
Services. Camden Fairview High School 
explains to both students and their parents how 
the NCRC relates to employment with industry 
partners. Rogers New Technology High School 
plans field trips to local employers to 
demonstrate alignment of the NCRC to “real, 
high-wage, high-growth positions within the 
community.” They also show ACT videos 
promoting the value of the NCRC even for 


students who plan to earn a college degree. In 
addition to explaining its importance, Western 
Yell assigns students to conduct internet 
research to learn about the meaning of the 
NCRC and its benefits. Most information is 
provided by the Arkansas Division of 
Workforce Services website, which includes 
brochures for job seekers and employers as 
well as lists of employers who recognize the 
NCRC in Arkansas and nationally.* 


The Lead Career Coach at Rivercrest High 
School explains the NCRC in a college and 
career readiness presentation, and he 
celebrates student success stories on social 
media. Faculty also receive professional 
development to better understand the value of 
the NCRC and the potential impact of 
WorkKeys Curriculum on general achievement, 
especially when reading or math is involved. 
Manila High School hosts guest speakers from 
local industries to talk about hiring 
requirements, including the NCRC. In another 
effort to highlight the value of the NCRC, 
students receive their NCRCs during an 
awards ceremony at school. Similarly, students 
at Crossett High School are recognized with 
cords to wear at graduation, a certificate, and a 
cash award for students who earn an NCRC at 
the Silver level or higher. 


Warren High School is situated in an ACT 
Work Ready Community, which is an area 
where economic developers, employers, and 
educational institutions partner to measure 
skills gaps using WorkKeys and plan workforce 
development efforts.° The local ACT Work 
Ready Team meets monthly and plans 
information-sharing activities throughout the 
community. For students, this includes guest 
speakers, promotional banners, recruiting by 
the Internship/Youth Apprenticeship Program 
Coordinator, and instructors promoting the 
NCRC and assisting students throughout the 
year. Warren High School also fosters friendly 
competitions and provides rewards and 
celebrations for students who earn an NCRC. 
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WorkKeys Recognition 


Ultimately, the value of an NCRC to an 
individual depends on whether employers and 
educational institutions require or recognize 
WorkKeys performance when making 
decisions about hiring or admission. Survey 
results indicate varying degrees of WorkKeys 
recognition. Western Yell High School 
indicated that local employers do not recognize 
WorkKeys, but students are informed about 
employers across the state who do. 
Russellville High School reported that “a few” 
local manufacturers recognize the NCRC, and 
Manila High School said that WorkKeys is 
recognized by “several” employers in the 
county and internship programs sponsored by 
local postsecondary institutions. In the area 
surrounding Warren High School, businesses 
are “starting to recognize WorkKeys and 
NCRC when hiring new employees.” Some 
employers have established minimum 
WorkKeys scores through job profiling. 
Crossett High School noted that WorkKeys is 
recognized by the Federation for Advanced 
Manufacturing Education, the University of 
Arkansas at Monticello College of Technology 
— Crossett, and several local employers. 


Benton High School provided a list of 16 
employers in the county who recognize 
WorkKeys. Rogers New Technology High 
School reported that local employers, 
especially advanced manufacturing firms, use 
WorkKeys to ensure quality candidates and 
support employee retention. According to Pine 
Bluff High School, more than 50 local 
employers recognize WorkKeys. Camden 
Fairview High School indicated that “all” local 
industries require WorkKeys, and they gave 
several examples (Lockheed Martin, Aerojet, 
and General Dynamics). Finally, Rivercrest 
High School provided a list of 77 local 
employers who recognize WorkKeys. Also, a 
local college recognizes WorkKeys by offering 
three credit hours for the “Workplace 
Essentials” course, which is required for most 


CTE programs of study. The college also offers 
monetary incentives of $25—-$100 depending 
on the NCRC level achieved. 


Conclusions 


A brief survey was administered to Arkansas 
high schools with greater-than-expected 
WorkKeys performance with the goal of 
identifying potentially effective practices for 
promoting career readiness. All respondents 
indicated that WorkKeys Assessments were 
required components of courses designed to 
prepare students for college and careers. The 
high schools offered a variety of additional 
strategies to foster career readiness. This 
included internships, work-based learning, or 
youth apprenticeship programs as well as 
integrating career readiness curricula into CTE 
courses, active counseling to support the 
development of students’ career plans, project- 
based learning across the curriculum, and 
interaction with local business leaders. All high 
schools reported using WorkKeys Curriculum 
to help prepare students to take the NCRC 
assessments, and some schools use 
WorkKeys Curriculum to determine when 
students are ready to take the tests. Only one 
school described classroom instruction 
specifically targeted at the skills measured by 
the NCRC assessments, but general academic 
coursework should help students prepare for 
WorkKeys as well. The high schools described 
a variety of methods to increase awareness of 
the importance of performing well on 
WorkKeys, including informational 
presentations, videos, and assignments, as 
well as guest speakers, field trips, motivational 
banners, and public recognition for NCRC 
attainment. High schools in Arkansas and 
around the US can learn from each other to 
support the shared goals of closing skills gaps, 
ensuring career readiness and success, and 
supporting economic development. 
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Appendix: Statistical Analysis 
Data 


ACT Research extracted WorkKeys data from Arkansas examinees from 2017 
through 2019 for the three NCRC assessments: Applied Math, Workplace 
Documents, and Graphic Literacy. To identify high school testers, the WorkKeys data 
were merged with Arkansas statewide ACT testing data from the 2016-2017, 2017— 
2018, and 2018-2019 academic years. Schools were included in the sample if they 
had 15 or more students who took Applied Math, Workplace Documents, and 
Graphic Literacy and had ACT scores. The merged data set included 2,534 students 
from 48 high schools. Statewide ACT testing occurs in spring of junior year, and the 
typical time between the ACT and WorkKeys was 9-10 months, which means that 
most students took WorkKeys as high school seniors. The sample was 53.2% 
female, 19.5% Black/African American, 12.4% Hispanic/Latino, and 62.0% White. 


Analysis 


Descriptive analyses were first carried out to examine the distributions of WorkKeys 
scores, ACT scores, and GPA as well as their correlations. Next, multi-level models 
were fit to the data to evaluate academic achievement and demographic variables as 
predictors of WorkKeys scores. In these analyses, ACT math predicted Applied 
Math, ACT reading predicted Workplace Documents, and ACT science predicted 
Graphic Literacy. Whereas the descriptive analyses grouped all students together, 
the multi-level models accounted for the nesting of students within schools. Models 
were fit with combinations of student-level and school-level predictors including 
ethnicity (White or Non-White), gender, and ACT score. The full model is represented 
by the following equations: 


WK; = Boj + B,;NonWhite;; + 62 ;Female;; + B3jACT;; + Nj 


Boj = Yoo + Yo1Prop_NonWhite; + ¥o2Prop_Female; + Y¥93ACT; + uo; 


Bij = Y10 
Bj = Y20 
Bsj = 730 


In this model, the WorkKeys score for student / in school j is predicted by ethnicity, 
gender, and ACT score. Each school intercept £9; is predicted by proportion non- 
White, proportion female, and mean ACT score. Note that entering the same 
predictors at both levels eliminates potential bias in the student-level coefficients 
caused by associations between school-level predictors and f;. In exploratory 
analyses, the coefficients within schools were allowed to vary, but that variation was 
not statistically significant. Thus, the coefficients for non-White, gender, and ACT 
score were held constant across schools. The school-level random effect uo; served 
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as an indicator of average WorkKeys performance relative to expected. A positive 
value would indicate that, on average, students in a certain school earned higher 
WorkKeys scores than students in other schools controlling for all predictor variables. 


Descriptive Statistics 


Table 1 shows summary statistics for the distributions of student achievement 
variables. WorkKeys scale scores range from 65—90, and they are transformed to 
level scores with a maximum of 7. The average WorkKeys scores in Table 2 
correspond to level 4 on Applied Math (76—79), level 4 on Workplace Documents 
(77-80), and level 5 on Graphic Literacy (78-81). WorkKeys level scores are tied to 
descriptions of specific workplace knowledge and skills, and they are used to award 
the NCRC. Specifically, examinees must attain Level 3 or higher on the Applied 
Math, Workplace Documents, and Graphic Literacy to earn Bronze, 4 or higher to 
earn Silver, 5 or higher to earn Gold, and 6 or higher to earn Platinum. More than 
90% of the sample earned an NCRC, with 28.5% earning Bronze, 33.0% Silver, 
20.1% Gold, and 9.8% Platinum. 


The ACT is an academic achievement test commonly used to support college 
admissions decisions. The average ACT scores in Table 1 suggest that the sample 
had slightly lower average achievement than the national 2018 high school 
graduating class. For comparison, the national averages were 20.5 for ACT math, 
21.3 for ACT reading, and 20.7 for ACT science.® 


Table 1. Student Achievement Variable Distributions 


: 25th : 75th 
arabe Percentile megan cannes 
WorkKeys Applied Math 79.2 4.6 76.0 79.0 82.0 
WorkKeys Workplace Documents 79.0 4.0 77.0 80.0 82.0 
WorkKeys Graphic Literacy 79.2 4.1 76.0 79.0 82.0 
ACT Math 18.3 4.0 15.0 17.0 20.0 
ACT Reading 18.8 5.5 14.0 18.0 22.0 
ACT Science 18.7 4.5 16.0 18.0 22.0 


Figure 1 illustrates the distributions of variables used in subsequent analyses. 
Specifically, histograms of Applied Math, Workplace Documents, Graphic Literacy, 
ACT math, ACT reading, and ACT science are shown in the diagonal entries. Scatter 
plots showing the bivariate relationships are below the diagonal, and correlations 
coefficients are above the diagonals. All correlation coefficients were moderate to 
strong in magnitude and statistically significant (p < .001). Correlations among 
WorkKeys scores ranged from .64 to .72, correlations among ACT scores ranged 
from .64 to .74, and correlations between WorkKeys and ACT ranged from .56 to .71. 
Overall, students who demonstrated higher achievement on one measure tended to 
demonstrate higher achievement on all others. 
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Figure 1. Histograms, Scatter Plots, and Correlations for WorkKeys, ACT, and GPA. 
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Results 


Tables 2, 3, and 4 show the multi-level model parameter estimates and indicators of 
statistical significance. Model 1, which is known as the “unconditional” model, serves 
a point of reference for the subsequent models. Across assessments, patterns in the 
statistical significance of model parameters were generally similar, so only the 
Applied Math models are described in detail. Model 2 added non-White and female 
as student-level predictors, and this had the effect of reducing the between-school 
variance (Too) and within-school variance (o2). However, the addition of ACT math 
score as a student-level predictor in Model 3 reduced those variances substantially. 
Put another way, ACT math scores accounted for much more variance in Applied 
Math scores than the non-White and female indicators. This was apparent from the 
increase in R? from .156 to .551. Model 4 investigated school-level predictors by 
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themselves, and only mean ACT math scores had a statistically significant 
coefficient. Model 5 included all student-level and school-level predictors. Only the 
statistically significant coefficients from Model 5 were retained in the final model 
(Model 6). Model 6 included only the following coefficients: student non-White 
indicator and student ACT math score. Note that the between-school and within- 
school variances were nearly identical in Model 5 and Model 6. The same Model 6 
parameterization was applied for Workplace Documents and Graphic Literacy. 


Table 2. Applied Math Model Parameter Estimates 


Estimate Model 1 I CoXe (=) 74 LM Coxe (=) Is) Model 4 NM CoXe (=) Ie) NM Coxe (=) MCs) 
Intercept Yoo 19.202*** 80.201*** 65.319*** 64.981*** 65.318*** 65.209*** 
Non-White Bajly10 -2.070*** -1.001*** -1.020*** -1.014*** 
Female Bl y20 -0.515** -0.175 -0.175 

ACT Math Bail ¥30 0.782*** 0.783*** 0.784*** 
Proportion Non-White yo1 -0.804 0.213 

Proportion Female You -0.168 0.043 

Mean ACT Yo3 0.797*** -0.005 

Too 2.189 1.716 0.524 0.417 0.568 0.524 
o? 18.630 17.844 9.461 18.601 9.463 9.465 
R? .120 .156 5511 115 .552 551 


Table 3. Workplace Documents Model Parameter Estimates 


Estimate WM Coke (=1 i WY Cote (1 7 Model 3 Model 4 LY Coke (=9 ft) Model 6 
Intercept Yoo 79.168*** 79.208*** C120 69.686*** 69.835*** 71.455*** 
Non-White Bij/yi0 -1.300*** -0.506*** -0.494*** -0.458** 
Female B2;/y20 0.796*** 0.504*** 0.5077" 

ACT Reading B3j/¥30 0.416*** 0.414*** 0.419*** 
Proportion Non-White Yo1 -0.511 0.004 

Proportion Female Yo2 -0.034 -0.513 

Mean ACT Reading Y03 0.514*** 0.092 

T00 2.365 2.089 1.171 1.071 1.207 1.167 
o2 14.039 13.655 9.255 14.046 9.257 9.313 


R2 156 .180 443 154 443 .439 
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Table 4. Graphic Literacy Model Parameter Estimates 


Estimate WM Coke (=) Bia WM Coke C=) 4 Model 3 Model 4 YoYo (=) be) NM Coxe (=) MCs) 
Intercept Yoo 79.268*** 79.809*** 68.976*** 68.877*** 69.105*** 69.016*** 
Non-White Bajly10 -1.499*** -0.454** -0.438** -0.448** 
Female Bail y20 -0.032 0.075 0.094 

ACT Science B3ily30 0.556*** 0.555*** 0:559°** 
Proportion Non-White Yo1 -0.52 -0.057 

Proportion Female Yo2 -1.724 -1.748 

Mean ACT Science Yo3 0.611*** 0.043 

Too 2.000 1.666 0.729 0.588 0.720 0.725 
o? 14.986 14.637 9.326 14.987 9.328 9.324 
R? 131 .152 .458 127 .458 458 


Model parameter estimates allowed for estimation of the school-level random 
effects (uo;), which indicated average WorkKeys performance relative to expected. 
Each school had three residuals, one for each WorkKeys Assessment (Table 5). 
Those residuals were averaged to identify schools with greater-than-expected 
WorkKeys scores (see Mean Random Effect in Table 5). The mean random effects 
ranged from 0.6 to 1.4 on the WorkKeys scale. While those differences may appear 
small, they are not so small relative to the standard deviations of WorkKeys score 
distributions (Table 1). Indeed, they represent effect sizes ranging from 
approximately 0.13 to 0.32. That is, for example, one school had average WorkKeys 
scores 0.32 standard deviations higher than the average for schools enrolling 
students with similar ACT scores and demographics. The maximum effect sizes on 
Applied Math, Workplace Documents, and Graphic Literacy were 0.37, 0.41, and 
0.42 standard deviations, respectively. The 10 schools in Table 5 (with the 10 
highest average random effects) were contacted for this study. 
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Table 5. Random Effects 


Applied Math Workplace Graphic 
Documents Literacy 
Mean Mean Mean 
Percent Percent Random ACT Random ACT Random Random 
Non-White Female Mean Effect axexcleliale) Mean Effect sler(= ales) Mean Effect Effect 
82 39.0% 69.5% 17.9 80.9 1:7 18.7 80.9 1.6 18.4 80.0 0.8 1.4 
17 76.5% 47.1% 16.7 78.8 0.6 17.1 80.1 1.3 17.1 81.2 1.7 1.2 
42 35.7% 52.4% 18.9 80.9 0.8 19.9 81.0 1.2 19.6 80.6 O:7 0.9 
16 25.0% 25.0% 16.2 79.4 0.8 16.6 79.7 0.9 17.3 79.5 0.6 0.8 
84 20.2% 57.1% 19.6 80.9 0.4 21.4 81.1 0.7 20.6 81.6 1.1 0.7 
147 57.8% 44.2% 18.7 80.0 0.7 19.7 80.1 0.6 19.3 80.3 0.7 0.7 
95 17.9% 53.7% 20.6 81.7 0.4 21.1 81.1 0.8 21:0 81.4 0.7 0.7 
15 46.7% 60.0% 16.2 77.5 0.0 16.1 80.1 1.4 17.0 79.2 0.5 0.6 
44 11.4% 45.5% 18.6 79.9 0.1 17.4 79.8 1:0 18.0 79.9 0.7 0.6 


28 100.0% 42.9% 15.8 FEA 0.3 15.4 77.9 0.4 14.8 78.2 1.0 0.6 
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